PLT-2782 | Janus0.6 #3852

aarshi0301 · 2024-12-09T09:09:28Z

Change description

Description here

optimise cassandra calls
prefetch all the properties of vertex
prefetch all the properties of relation

Atlan PR : https://github.com/atlanhq/atlan/pull/6024/files

stats : https://docs.google.com/spreadsheets/d/1uRahRlM32EKlqOR5c_a4ZB72z9_deXN_CqBS1fbkL2Y/edit?gid=0#gid=0

Testing Notes

Validated contracts

Validated McMonitor

Validated Business Attributes

Validated Columns

Local

Table

"entities": [
        {
            "typeName": "Table",
            "attributes": {
                "lastSyncRunAt": 0,
                "connectionQualifiedName": "default/snowflake/1",
                "__modifiedBy": "admin",
                "columns": [
                    {
                        "guid": "3209048c-bfaf-4304-b6fe-05ec47801f0c",
                        "typeName": "Column",
                        "attributes": {
                            "popularityScore": 1.17549435E-38,
                            "connectorName": "snowflake"
                        },
                        "uniqueAttributes": {
                            "qualifiedName": "table304"
                        }
                    }
                ],
                "__hasLineage": false,
                "adminUsers": [
                    "aarshi",
                    "suman"
                ],
                "queryPreviewConfig": {
                    "env": "[test1, test2]"
                },
                "ownerGroups": [],
                "sourceUpdatedAt": 0,
                "BNEa1vMTtmgZqkTh2icsJB.B8vVPtZI9lbGokUtFjcft4": [
                    "cql",
                    "sql",
                    "test1",
                    "tset2"
                ],
                "announcementUpdatedAt": 0,
                "__timestamp": 1733825739630,
                "sourceCreatedAt": 0,
                "adminRoles": [],
                "adminGroups": [],
                "qualifiedName": "table304",
                "__state": "ACTIVE",
                "sourceReadRecentUserRecordList": [
                    {
                        "typeName": "PopularityInsights",
                        "attributes": {
                            "recordTotalUserCount": 0,
                            "recordUser": "ATLANADMIN",
                            "recordQueryCount": 0,
                            "recordWarehouse": null,
                            "recordComputeCost": 0.0,
                            "recordLastTimestamp": 1693979477505,
                            "recordQuery": null,
                            "recordQueryDuration": 0,
                            "recordMaxComputeCost": 0.0,
                            "recordComputeCostUnit": null
                        }
                    }
                ],
                "dataContractLatest": {
                    "guid": "cd896762-eb74-4742-9825-7c855811506f",
                    "typeName": "DataContract",
                    "attributes": {
                        "popularityScore": 1.17549435E-38
                    },
                    "uniqueAttributes": {
                        "qualifiedName": "table304/Table/contract/V1"
                    }
                },
                "meanings": [],
                "__guid": "ec97889d-f07d-4ec7-98c6-a8badbb79b0d",
                "__createdBy": "admin",
                "__modificationTimestamp": 1733825834082,
                "name": "table304",
                "certificateUpdatedAt": 0
            },
            "guid": "ec97889d-f07d-4ec7-98c6-a8badbb79b0d",
            "status": "ACTIVE",
            "displayText": "table304",
            "classificationNames": [],
            "classifications": [],
            "meaningNames": [],
            "meanings": [],
            "isIncomplete": false,
            "labels": [],
            "createdBy": "admin",
            "updatedBy": "admin",
            "createTime": 1733825739630,
            "updateTime": 1733825834082
        }
    ],

MCMonitor

"entities": [
        {
            "typeName": "MCMonitor",
            "attributes": {
                "sourceCreatedAt": 0,
                "lastSyncRunAt": 0,
                "adminRoles": [],
                "__modifiedBy": "admin",
                "adminGroups": [],
                "qualifiedName": "monitor1",
                "__state": "ACTIVE",
                "__hasLineage": false,
                "sourceReadRecentUserRecordList": [],
                "meanings": [],
                "__guid": "01b22fa6-0fb3-490f-a6aa-b51d14906297",
                "adminUsers": [],
                "mcMonitorRuleScheduleConfig": {
                    "typeName": "MCRuleSchedule",
                    "attributes": {
                        "mcRuleScheduleType": null,
                        "mcRuleScheduleStartTime": 0,
                        "mcRuleScheduleCrontab": null,
                        "mcRuleScheduleIntervalInMinutes": 0
                    }
                },
                "ownerGroups": [],
                "sourceUpdatedAt": 0,
                "BNEa1vMTtmgZqkTh2icsJB.B8vVPtZI9lbGokUtFjcft4": [],
                "__createdBy": "admin",
                "announcementUpdatedAt": 0,
                "__modificationTimestamp": 1733739725460,
                "name": "monitor1",
                "certificateUpdatedAt": 0,
                "__timestamp": 1733739725460
            },
            "guid": "01b22fa6-0fb3-490f-a6aa-b51d14906297",
            "status": "ACTIVE",
            "displayText": "monitor1",
            "classificationNames": [],
            "classifications": [],
            "meaningNames": [],
            "meanings": [],
            "isIncomplete": false,
            "labels": [],
            "createdBy": "admin",
            "updatedBy": "admin",
            "createTime": 1733739725460,
            "updateTime": 1733739725460
        }
    ]

Type of change

Bug fix (fixes an issue)
New feature (adds functionality)

Related issues

Fix #1

Checklists

Development

Lint rules pass locally
Application changes have been tested thoroughly
Automated tests covering modified code pass

Security

Security impact of change has been considered
Code follows company security practices and guidelines

Code review

Pull request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary
"Ready for review" label attached and reviewers assigned
Changes have been reviewed by at least one other contributor
Pull request linked to task tracker where applicable

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java

sriram-atlan and others added 12 commits December 6, 2024 01:20

janus0.6 changes

4b00c08

fix object type and simplify conditional logic

af722cb

better approach

4f1289b

fallback always

74858aa

add log to track non null attributes

d237e9a

add log to understand relationship labels

d1e72e0

use prefetch final for map,struct and array of primitives

837d507

fallback on relationship links only

5c5bb2d

preload properties for relatedvertex

62c4f97

remove check for maps

bf34539

fix business attributes

f6c9bde

fix log level

664501f

aarshi0301 self-assigned this Dec 9, 2024

aarshi0301 added the master label Dec 9, 2024

aarshi0301 marked this pull request as draft December 9, 2024 09:10

update branch

87568ed

aarshi0301 marked this pull request as ready for review December 9, 2024 12:00

aarshi0301 requested review from mehtaanshul, sumandas0, hr2904, sriram-atlan and nikhilbonte21 December 9, 2024 12:59

aarshi0301 changed the title ~~Janus0.6~~ PLT-2782| Janus0.6 Dec 10, 2024

aarshi0301 changed the title ~~PLT-2782| Janus0.6~~ PLT-2782 | Janus0.6 Dec 10, 2024

mehtaanshul reviewed Dec 10, 2024

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Show resolved Hide resolved

mehtaanshul reviewed Dec 10, 2024

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Show resolved Hide resolved

aarshi0301 added 2 commits December 10, 2024 13:52

add strict check for business attribute

291cb2e

code cleanup

1078182

sriram-atlan reviewed Dec 10, 2024

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Show resolved Hide resolved

compute flag only if property exists

dce4f05

verbose comment

b2a58a6

mehtaanshul previously approved these changes Dec 10, 2024

View reviewed changes

sriram-atlan reviewed Dec 10, 2024

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Outdated Show resolved Hide resolved

sriram-atlan reviewed Dec 10, 2024

View reviewed changes

repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Outdated Show resolved Hide resolved

add policy variables

0da7195

aarshi0301 dismissed mehtaanshul’s stale review via 0da7195 December 10, 2024 13:54

fix pipeline

759c230

sriram-atlan previously approved these changes Dec 10, 2024

View reviewed changes

remove log line

f2de42d

aarshi0301 dismissed sriram-atlan’s stale review via f2de42d December 10, 2024 15:31

sriram-atlan approved these changes Dec 10, 2024

View reviewed changes

sriram-atlan merged commit 6e1f8e9 into master Dec 10, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PLT-2782 | Janus0.6 #3852

PLT-2782 | Janus0.6 #3852

aarshi0301 commented Dec 9, 2024 •

edited

Loading

PLT-2782 | Janus0.6 #3852

PLT-2782 | Janus0.6 #3852

Conversation

aarshi0301 commented Dec 9, 2024 • edited Loading

Change description

Testing Notes

Type of change

Related issues

Checklists

Development

Security

Code review

aarshi0301 commented Dec 9, 2024 •

edited

Loading